A New Communication and Computation Overlapping Model with Loop Sub-Partitioning and Dynamic Scheduling

The latency hiding techniques can significantly improve the performance of the parallel programs in distributed memory systems. This paper presents a communication and computation overlapping model to hide the communication latency in data parallel programs. The communication and computation overlapping model makes use of the loop subpartitioning scheme in which a given loop partition is partitioned again into the finer partitions called loop subpartitions. One of the loop sub-partitions can be effectively overlapped with communication which is incurred by the rest of the loop sub-partitions. This loop sub-partitioning scheme can hide latency effectively within the scope of a single loop, while the previous latency hiding techniques require several independent loops or computations. Communication incurred by each loop sub-partition has dynamic behavior, so it is difficult to predict the completion time in MIMD machines. In our model, the loop subpartitions are dynamically scheduled according to the completion of their communication. This dynamic scheduling technique is based on the dependence graphs which represent the relationship between message receptions and loop sub-partitions. The preliminary results for IBM SP2 with LAPI show that our model significantly improves the performance.